Sparse, Dense, and Attentional Representations for Text Retrieval

Abstract Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document its inner product with the query. We investigate capacity of this architecture relative to sparse bag-of-words models attentional neural networks. Using both theoretical empirical analysis, we establish connections between dimension, margin gold lower-ranked documents, length, suggesting limitations in fixed-length encodings support precise long documents. Building on these insights, propose a simple model that combines efficiency dual some expressiveness more costly architectures, explore sparse-dense hybrids capitalize precision retrieval. These outperform strong alternatives large-scale